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1.  Introduction 


This  project  is  concerned  with  the  study  of  advanced 
techniques  for  the  analysis  of  reconnaissance  imagery.  It 
is  being  conducted  under  Contract  DAAG-53-76C-0138  (DARPA 
Order  3206),  monitored  by  the  U.S.  Army  Night  Vision  Laboratory, 
Ft.  Belvoir,  VA  (Dr.  George  Jones).  The  Westinghouse  Systems 
Development  Division,  as  a  subcontractor,  is  investigating 
implementation  of  the  techniques  being  developed  by  Maryland; 
the  subcontract  is  entitled  "Architecture  for  higher  level 
digital  image  processing".  Dr.  Glenn  E,  Tisdale  is  program 
manager  for  Westinghouse,  and  Dr.  Azriel  Rosenfeld  is  principal 
investigator  at  the  University  of  Maryland. 

The  current  phase  of  this  project,  initiated  in  April  1978, 
is  a  continuation  of  a  project  entitled  "Algorithms  and  Hardware 
Technology  for  Image  Recognition"  (May  19 76 -March  197  8)  .  The 
earlier  project  [1]  was  concerned  primarily  with  tactical 
target  detection  on  forward-looking  infrared  (FLIR)  imagery. 
Specific  efforts  involved  image  modeling,  smoothing,  noise 
cleaning,  edge  detection  and  thinning,  thresholding,  tracking, 
feature  extraction,  and  classification.  Through  the  use  of 
convergent  evidence,  based  on  coincidences  between  edge  maxima 
and  borders  of  above-threshold  regions,  excellent  object  extrac 
tion  performance  was  achieved.  Westinghouse  studied  the  CCD 
implementations  of  many  of  the  algorithms  that  were  developed, 


and  breadboarded  one  basic  function,  a  sorter.  Communication 
among  the  Maryland,  Westinghouse ,  and  NVL  groups  was  very  good 
and  lead  to  greatly  accelerated  transfer  of  advanced  image 
understanding  techniques. 

The  current  project  is  concerned  with  the  development  and 
application  of  advanced  techniques  for  image  processing, 
feature  detection,  segmentation,  texture  and  shape  analysis, 
and  region  representation.  (The  "overlays"  in  the  project 
title  refer  to  region  representations  which  specify  background 
areas  or  possible  object  locations.) 

Section  2  of  this  report  summarizes  accomplishments  on  the 
project  during  the  past  six-month  period.  Summaries  for  earlier 
periods  can  be  found  in  previous  Semiannual  Reports  [2-4];  these 
reports  also  appear  as  Project  Status  Reports  in  the  semiannual 
DARPA  Image  Understanding  Workshops  [5-8] . 

The  project  is  expected  to  continue  for  an  additional 
two-year  period,  under  the  title  "Understanding  Features, 
Objects,  and  Backgrounds".  Section  3  summarizes  the  proposed 
efforts  to  be  conducted  during  this  period. 

Section  4  lists  the  individual  reports  issued  and  papers 
published  during  the  current  phase  of  the  project,  arranged 
by  type  and  subject.  The  contents  of  the  Quarterly  Reports  on 
the  Westinghouse  subcontract  [9-15]  are  summarized  in  the 
Appendix. 


2 .  Semiannual  Report  for  the  period  1  October  1979-30  March  1980 

In  this  section,  activities  on  the  project  during  the  past 
six  months  are  reviewed  under  three  headings:  (1)  segmentation 
and  texture  analysis;  (2)  local  and  global  shape  analysis; 

(3)  hierarchical  representation.  Numbers  in  brackets  refer 
to  the  technical  reports  listed  in  Section  4 . 

2 . 1  Segmentation  and  texture  analysis 

A.  Edge  detection  , 

Edges  are  generally  detected  by  thresholding  the  output 
of  some  type  of  difference  operator;  but  the  choice  of  a 
threshold  for  this  purpose  is  not  easy,  since  the  histogram  of 
difference  values  tends  to  fall  off  smoothly  from  a  peak  near 
zero.  Threshold  selection  becomes  easier  if  we  suppress  non¬ 
maximum  difference  values  (in  the  gradient  direction)  before 
histogramming.  As  Figure  1  shows,  this  yields  a  histogram  com¬ 
posed  of  a  sharpened  peak  near  zero  together  with  small  sets 
of  higher  values;  the  latter  are  likely  to  be  good  choices  for 
edge  points  [67] . 

Difference  operators  for  edge  detection  can  be  designed 
by  fitting  a  polynomial  surface  to  the  gray  levels  in  the 
neighborhood  of  a  point,  and  taking  the  gradient  of  that  sur¬ 
face  as  an  estimate  of  the  image  gradient.  This  approach  can 
be  generalized  to  the  design  of  operators  for  surface  detection 
in  three-  (or  higher-)  dimensional  arrays  of  data,  such  as 


d 


Figure  1 


e 

Nonmaximum  suppression  as  an  aid  in  edge  detection, 
(a)  Image;  (b)  digital  (Sobel)  gradient  magnitudes; 
(c)  results  of  suppressing  nonmaxima  in  the  gradient 
direction;  (d)  results  of  thresholding  (b)  at  6; 

(e)  histograms  of  (b)  and  (c)  superimposed. 


those  obtained  by  reconstructing  objects  from  x-ray  projections, 
by  stacking  cross-sections,  or  by  stacking  successive  frames 
in  a  time  sequence  of  images.  Surface  detection  provides  re¬ 
sults  that  are  more  accurate  and  more  reliable  than  those 
obtained  by  applying  two-dimensional  edge  detectors  to  the 
individual  slices,  as  can  be  seen  from  Figure  2.  This  work 
is  part  of  a  Ph.D.  thesis  on  processing  and  segmentation  of 
three-dimensional  arrays . 

B .  Pixel  classification  and  texture  analysis 
During  the  past  reporting  period,  an  M.S.  thesis  was  com¬ 
pleted  [55]  on  a  general-purpose  software  package  for  performing 
relaxation  operations  on  arrays  of  pixels.  This  package  allows 
the  user  to  specify  the  process  for  computing  initial  probabi¬ 
lities,  the  neighborhood  to  be  used,  and  the  probability  ad¬ 
justment  algorithm  (including  the  compatibility  coefficients) . 

As  an  application,  a  light/dark  relaxation  process  was  imple¬ 
mented;  examples  of  this  process  can  be  found  in  earlier  status 
reports  [3,6].  Some  analytical  results  regarding  such  two-]abel 
relaxation  processes  can  be  found  in  the  sixth  quarterly  report 
on  the  Westinghouse  subcontract  [l 4]  - 

Relaxation  has  been  successfully  used  to  improve  pixel 
classification  based  on  color,  as  reported  elsewhere.  It  can 
similarly  be  used  to  improve  pixel  classification  in  single¬ 
band  images  based  on  local  property  values  such  as  gray  level 
and  "busyness".  Figure  3a  shows  a  house  picture  containing 


a 


b 


c 


Figure  2  (see  next  page  for  caption) 
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Figure  2.  Surface  detection  in  3-d  arrays.  (a-c)  Three  consecutive 
cross-sections  of  a  CT  reconstruction;  (d)  results  of 
applying  the  2-d  Prewitt  operator  to  the  middle  cross-section 
(e)  results  of  applying  a  3-d  Prewitt  operator  to  the  three 
cross-sections . 


five  principal  types  of  regions — sky,  grass,  bushes,  brick, 
and  shadow.  The  bush  and  shadow  classes  are  very  difficult 
to  distinguish;  they  have  similar  mean  vectors,  and  the  bush 
class  is  more  variable,  so  that  a  maximum-likelihood  classifi¬ 
cation  (based  on  Gaussian  fitting  to  the  clusters  defined  by 
hand  segmentation)  misclassif ies  most  of  the  shadow  pixels  as 
bush  (Figure  3b) .  The  results  are  greatly  improved  when  re¬ 
laxation  is  used  to  adjust  the  initial  class  probabilities 
for  each  pixel  based  on  those  of  its  neighbors;  see  Figure  3c. 
Similar  improvement  is  obtained  when  the  busyness  values  are 
iteratively  smoothed,  e.g.  by  median  filtering,  prior  to 
clustering  and  classification  (Figure  3d) .  Further  details  on 
these  experiments  can  be  found  in  [65] . 

Iterative  smoothing  can  also  be  used  to  improve  the  results 
of  texture  classification  using  texture  features  derived  from 
small  windows,  as  described  in  [56] . 

C .  Iteractive  segmentation 

An  interactive  image  segmentation  system  is  being  designed 
as  a  contribution  to  the  DARPA/DMA  Testbed.  The  system  allows 
the  user  to  designate  samples  of  two  classes  (e.g.,  objects 
and  background) .  It  analyzes  the  samples,  designs  a  classi¬ 
fier  to  discriminate  them,  and  displays  the  classification 
results  to  the  user  for  evaluation;  if  errors  are  designated, 
the  system  attempts  to  modify  the  classifier  so  as  to  eliminate 
them.  The  user  need  not  know  anything  about  the  classification 
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process  or  the  features  that  are  used  for  classification;  the 
system  selects  them  from  a  prespecified  repertoire.  The  cur¬ 
rent,  pilot  version  of  the  system  classifies  pixels  based  on 
gray  level  only;  future  versions  will  make  use  of  various  types 
of  local  features  and  will  allow  more  than  two  classes. 

D.  Mosaicking 

A  relaxation  method  for  eliminating  seams  from  photomosaics 
without  degrading  image  detail  is  described  in  [72] . 


2 . 2  Local  and  global  shape  analysis 


A.  Corner  detection 

Several  types  of  operators  have  been  developed  that  re¬ 
spond  to  the  presence  of  "corners"  (i.e.,  sharp  changes  in  edge 
direction)  in  an  unsegmented  image  [69]  .  For  example,  one  can 
express  the  rate  of  change  in  the  gradient  direction  in  terms 
of  first  and  second  derivative  operators;  or  one  can  simply 
compute  a  digital  gradient  direction,  and  estimate  its  rate  of 
change  at  P  by  comparing  it  with  the  directions  at  the  appro¬ 
priate  neighbors  of  P.  To  measure  "cornerity",  the  rate  of 
change  in  gradient  direction  should  be  multiplied  by  the  gra¬ 
dient  magnitude,  since  we  are  only  interested  in  corners  that 
lie  on  edges.  Figure  4  shows  a  display  of  cornerity  values  for 
a  simple  grayscale  image;  the  results  seem  reasonable. 

B .  Collinearity  and  parallelism 

Collinear  and  parallel  (or  "antiparallel")  sets  of  edge  and 
line  segments  are  important  elements  in  the  description  of 
many  types  of  scenes.  The  following  paragraphs  describe  general 
purpose  programs  for  analyzing  collinearity  and  parallelism.  A 
more  specialized  program  that  links  edge  segments  based  on  gray 
level,  as  well  as  geometric,  criteria,  with  application  to  the 
detection  of  buildings  and  roads  on  aerial  imagery,  will  be 
described  in  a  forthcoming  report;  see  [85]. 

The  "collinearity  strength"  of  two  segments  depends  on 


several  factors : 


Figure  4.  Corner  detection  in  grayscale  images,  (a)  Image; 
(b)  results  of  "comerity"  computation. 
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Fig  are  5  (see  next  page  for  caption) 


CO 


(a)  The  distance  between  their  nearer  ends,  relative  to 
their  lengths 

(b)  The  angles  that  they  make  with  the  line  joining  their 
nearer  ends 

(c)  The  distance  between  their  farther  ends,  relative  to 
the  nearer-end  distance  and  lengths. 

A  collinearity  strength  measure  based  on  a  combination  of 
these  factors  gives  generally  reasonable  results,  as  illustrated 
in  Figure  5  [67] . 

Collinear  segments  can  be  grouped  into  "clusters"  based  on 
their  relative  sizes  and  separations.  Several  types  of  cluster 
merit  functions  can  be  used  for  this  purpose;  a  good  figure  of 
merit  should  depend  on  both  the  segment  density  and  the  total 
segment  length  in  the  given  cluster.  Examples  of  clusters 
defined  by  maximizing  such  a  figure  of  merit  are  given  in 
Figure  6.  For  a  report  on  these  experiments  see  [70]. 

Segments  can  also  be  linked  based  on  parallelism  (or,  in 
the  case  of  edge  segments,  antiparallelism:  the  dark  sides  of 
the  edges  should  face  in  opposite  directions) .  The  figure  of 
merit  for  this  linking  process  should  depend  on  the  separation 
of  the  segments ,  their  lengths  and  the  amount  by  which  they 
overlap,  as  well  as  their  parallelism  (i.e.,  the  angle  between 
them) .  Mutually  best  pairs  based  on  this  merit  function  can  be 
linked,  and  the  process  can  then  be  repeated  with  the  linked  pairs 
eliminated.  For  examples  of  the  results  obtained  using  this 
approach  see  [57] . 
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Figure  6.  Collinear  clusters  of  segments.  The  input  is  repeated  on  each  line,  with  the  results  of 
clustering  shown  below  it  for  successively  higher  values  of  a  clustering  parameter.  For 
the  value  used  in  (d) ,  the  central  cluster  is  linked  together  and  no  other  segments  are 
linked. 


C.  The  medial  axis 


The  medial  axis  (MA)  of  a  set  S  is  defined  as  the  set  of 
centers  (and  radii)  of  the  maximal  "disks"  contained  in  S,  or 
equivalently,  as  the  set  of  points  of  S  whose  distances  from 
the  complement  S  are  local  maxima.  It  can  be  used  as  a  com¬ 
pact  representation  of  S,  and  can  also  serve  as  a  basis  for 
approximating  S  by  a  union  of  "generalized  ribbons"  (=  con¬ 
nected  arcs  of  MA  points,  with  radii  specified  by  a  "width 
function"  defined  along  each  arc) . 

The  MA  is  sensitive  to  noise,  i.e. ,  to  errors  in  extracting 
the  set  S;  thus  it  would  be  desirable  to  define  it  directly  for 
unsegmented  images.  This  can  be  done  using  a  "gray -weighted" 
concept  of  distance,  but  it  is  hard  to  reconstruct  the  image 
from  such  an  MA.  Another  possibility  (the  "SPAN":  Spatial 
Piecewise  Approximation  by  Neighborhoods)  is  to  approximate 
the  image  by  maximal  homogeneous  disks ,  but  the  approximation 
process  is  computationally  costly.  Still  another  alternative 
is  to  assign  an  MA  score  to  each  point  P  based  on  the  presence 
of  high  gradient  values  at  pairs  of  positions  symmetrically 
located  with  respect  to  P;  but  this  process  turns  out  to  be 
quite  sensitive  to  noise. 

A  more  robust  approach  to  defining  an  MA  for  unsegmented 
images  is  based  on  a  characterization  of  the  MA  of  a  set  S  in 
terms  of  shrinking  and  expanding  operations  performed  on  S. 

Let  S 1  denote  the  result  of  shrinking  S  (i.e.,  deleting  its 


border)  n  times,  and  similarly  let  S ^  denote  the  result  of 
expanding  S  n  times  (S  (n)  =  S (_n) ) .  Then  it  is  not  hard  to 
see  that  =  (S^  -  S^  is  ^he  se^.  0f  points  at 

distance  k  from  S,  so  that  USk  is  the  MA.  To  generalize  this 
to  unsegmented  images,  we  use  local  MIN  operations  instead  of 
shrinking,  and  local  MAX  operations  instead  of  expanding;  we 
can  then  define  the  "MMMAT "  (=  min-max  MAT)  as  ESk.  Examples 
of  such  MMMATs  are  shown  in  Figure  7;  for  further  details  see 
[61] .  Approximations  to  the  image  can  be  reconstructed  by 
using  only  points  having  strong  MMMAT  values,  and  k's  that 
make  strong  contributions  to  these  values.  For  examples  of 
such  reconstructions  see  Figure  8;  see  [73]  for  a  report  on 
this  work. 

D .  Shape  segmentation 

Various  types  of  shape  features,  such  as  protrusions  and 
intrusions,  can  be  detected  by  comparing  boundary  arcs  with 
their  chords;  for  example,  if  the  chord  is  much  shorter  than 
the  arc,  or  if  the  arc  does  not  lie  close  to  the  chord,  that 
arc  must  be  a  proptrusion  or  intrusion.  Suppose  that  we  measure 
various  arc-chord  figures  of  merit  (e.g.,  arc  length  divided 
by  chord  length,  or  area  between  arc  and  chord  divided  by 
squared  chord  length)  for  every  arc.  In  many  cases,  extrema 
of  such  figures  or  merit  correspond  to  arcs  that  are  natural 
"pieces"  of  the  shape,  as  illustrated  in  Figure  9.  However, 


(a)  (b) 

Figure  7.  The  min-max  medial  axis  transformation 
(MMMAT).  (a)  Images;  (b)  MMMATs . 


(a)  (b)  (c)  (d) 

Figure  8.  Reconstruction  from  the  MMMAT.  (a)  Original  images; 

(b-d)  reconstructions  from  the  one,  two,  and  three 
largest  increments  at  those  points  having  values  above 
the  25th  percentile  (189,  582,  226,  and  462  out  of  4096 
pixels,  in  the  four  cases). 


Figure  9.  Shape  segmentation  based  on  extrema  of  arc/chord  functions, 
(a)  Maxima  of  area/chord^;  (b)  maxima  of  arc/chord; 

(c)  negative  maxima  of  area. 


this  approach  sometimes  leads  to  segmentations  that  are  not 
intuitively  plausible,  since  the  extreme  depend  only  on  (e.g.) 
the  curve's  slopes  at  the  arc  endpoints,  and  not  on  the  shape 
of  the  arc  between  the  endpoints;  see  [59] . 

Work  on  shape  segmentation  using  relaxation,  described  in 
earlier  reports,  is  being  extended  to  handle  shapes  with  major 
occlusions  or  missing  parts;  the  results  will  be  described  in 
a  forthcoming  report. 


2 . 3  Hierarchical  representation 


A.  Quadtrees  and  hextrees 

The  quadtree  algorithms  developed  on  this  project  usually 
involve  locating  neighbors  of  a  given  block  in  the  image  by 
searching  the  tree  starting  from  the  corresponding  node.  A 
general  treatment  of  neighbor  finding  in  quadtrees,  including 
an  analysis  of  the  expected  computational  costs,  can  be  found 
in  [62]. 

Quadtrees  are  defined  on  the  basis  of  recursive  subdivi¬ 
sion  into  quadrants;  they  involve  square  blocks,  and  four 
blocks  of  a  given  size  constitute  a  block  of  the  next  larger 
size.  For  some  purposes  it  may  be  desirable  to  define  a  repre¬ 
sentation  based  on  hexagonal  rather  than  square  blocks,  since 
such  a  representation  would  be  less  sensitive  to  rotation. 
Hexagons  cannot  be  combined  to  form  exact  hexagons  of  a  larger 
size,  but  one  can  combine  seven  hexagons  into  a  "ragged"  hexagon, 
and  this  process  can  be  iterated,  as  illustrated  in  Figure  10. 

A  detailed  discussion  of  how  to  define  hexagonal  "pyramids"  in 
this  way  can  be  found  in  [54]. 

B .  Quadtree  shape  approximation 

When  a  region  is  represented  by  a  quadtree,  the  upper 
levels  of  the  tree,  corresponding  to  large  blocks  of  the  image, 
define  approximations  to  the  region.  These  approximations  can 
be  used  to  estimate  shape  properties  such  as  moments,  and  to 
speed  up  shape  matching  by  eliminating  gross  mismatches 


Figure  10.  Hierarchical  hexagonal  grid  (three  levels). 


rapidly  [58],  For  example,  the  coordinates  of  the  centroid 
of  a  shape  can  be  estimated  to  a  fraction  of  a  pixel  using 
quadtree  approximations,  as  illustrated  in  Figure  11.  This 
should  make  it  possible  to  track  moving  shapes  quite  accurately; 
even  though  the  quadtree  itself  changes  radically  when  a  shape 
is  shifted,  the  moment  approximations  remain  stable.  Similarly, 
the  approximations  can  be  used  to  determine  upper  and  lower 
bounds  on  the  mismatch  area;  thus  if  we  are  matching  a  given 
shape  Sj  against  a  collection  of  stored  shapes  S^,S^,...,  we 
can  eliminate  any  such  that  the  lower  bound  on  the  mismatch 

of  S .  with  S .  exceeds  the  upper  bound  on  the  mismatch  of  S . 

j  1  c  j 

with  some  other  shape.  This  error  bounding  process  is  illu¬ 
strated  in  Figure  12. 

C.  Hierarchical  image  processing  and  segmentation 

Extensive  work  is  now  in  progress  on  the  use  of  pyramid 
structures  for  image  processing  and  segmentation.  The  following 
are  some  of  the  chief  areas  of  investigation: 

a)  Iterated  local  convolution  operations  can  be  used  to 
produce  large-kernel  convolutions  having  almost  exactly 
Gaussian  kernels.  These  can  in  turn  be  combined  to  yield 
various  types  of  circular  or  elongated  center-surround 
operators  [63]. 

b)  Image  pyramids  can  be  defined  in  which  the  blocks  at 
each  level  overlap;  this  largely  negates  the  objections 
to  conventional  power-of-2  pyramids  on  grounds  of  shift 
sensitivity . 


Area  Centroid  coordinates 


Figure  11.  Approximating  the  centroid  of  a  shape  using  its  quadtree  representation.  (a)  Airplane. 

(b)  Black  nodes  at  each  level  of  the  quadtree  representation  of  (a) ,  displayed  as  black 
blocks,  (c)  Cumulative  number  of  nodes,  area,  and  centroid  coordinates  as  a  function  of 
level. 
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Figure  12.  Lower  and  upper  bounds  on  the  mismatch  when  two  airplanes  are  matched  to  themselves  and  to 

each  other  (L=level,  LB=lower  bound,  UB=upper  bound).  Note  that  at  level  5,  the  lower  bound 
on  the  mismatch  to  each  other  exceeds  the  upper  bounds  on  the  mismatches  to  themselves. 


c)  In  an  overlapped  pyramid,  by  associating  nodes  with 
their  most  similar  ancestors,  one  can  establish 
linked  clusters  of  nodes  representing  homogeneous 
regions;  this  facilitates  smoothing  or  segmentation 
of  the  regions. 

d)  Local  operations  in  a  pyramid  can  be  used  to  detect 
simple  types  of  objects  in  the  image,  and  to  extract 
these  objects  by  local  thresholding.  This  approach  was 
applied  to  blob-like  objects  in  an  earlier  report;  it 
has  now  been  extended  to  streak-like  objects  [60] . 

e)  Pyramids  can  be  used  to  define  quadtree  approximations 
to  an  image  ("Q-images),  based  on  the  concept  that  a 
block  is  subdivided  only  if  it  is  unhomo ge ne ou s . 

f)  The  use  of  Q-images  facilitates  segmentation  by 
thresholding,  since  the  peaks  in  the  histogram  of  a 
Q-image  (where  each  block  contributes  its  mean  gray 
level,  a  number  of  times  proportional  to  its  size) 
tend  to  be  sharper  and  more  cleanly  separated.  The 
histogram  is  further  improved  when  we  eliminate  small 
blocks,  since  these  tend  to  lie  on  region  borders.  Con- 
versly ,  if  we  histogram  only  the  small  blocks,  we  obtain 
a  unimodal  histogram  whose  mean  is  a  good  threshold 

[68] .  More  generally,  we  can  find  blocks  in  the  quad¬ 
tree  corresponding  to  peaks  in  the  histogram,  and 


apply  local  thresholds  in  the  vicinity  of  these 
blocks  to  extract  the  appropriate  regions  [66] . 
g)  Q-images  can  also  be  used  to  improve  edge  detection, 
based  on  establishing  correspondences  between  edges 
in  the  Q-image  and  edges  in  the  original  image  [64] . 
On  the  use  of  Q-images  as  aids  in  image  smoothing 
see  [71]  . 


3.  Plans 


This  section  summarizes  the  planned  Maryland/Westinghouse 
efforts  on  the  next  phase  of  the  project,  under  the  title 
"Understanding  features,  objects,  and  backgrounds"  (the  West- 
inghouse  subcontract  will  be  entitled  "Evaluation  and  real-time 
implementation  of  image  understanding  algorithms").  Section  3.1 
dicusses  the  importance  of  developing  advanced  image  analysis 
capabilities  for  reconnaissance  applications.  Section  3.2 
outlines  the  proposed  approach,  and  Section  3.3  is  a  Statement 
of  Work  summarizing  the  principal  tasks  to  be  undertaken. 


3.1 


The  problem 


The  proposed  research  seeks  significant  improvements  in 
the  processing  of  sensor  images  that  are  involved  in  both  recon¬ 
naissance  and  weapon  delivery.  Military  relevance  in  these 
areas  is  obvious;  however,  in  recent  months  the  demand  for 
progress  in  the  area  of  weapon  delivery  has  become  most  urgent. 
The  reason  is  that  several  emerging  military  programs  are  now 
predicated  on  the  near-term  availability  of  semi-autonomous 
or  autonomous  target  acquisition  and  recognition  capability. 
Examples  are  the  Air  Force  LANTIRN  program  and  the  application 
°f  f ire-and-forget  performance  to  Army  missiles  and  large 
caliber  projectiles.  The  decision  by  the  services  to  place 
this  degree  of  reliance  on  image  recognition  appears  to  weigh 
favorably  the  major  improvements  in  mission  performance  which 
might  be  realized  with  its  use  against  its  present  admittedly 
developmental  status. 

The  performance  of  present  image  recognition  algorithms, 
which  is  to  some  extent  a  direct  result  of  earlier  work  on  this 
Maryland/Westinghouse  program,  is  probably  adequate  for  scenarios 
which  are  limited  in  range  and  scene  content.  In  more  demanding 
situations  it  will  be  lacking.  But  for  any  situation  improved 
recognition  performance  will  translate  directly  into  the  in¬ 
creased  probability  of  mission  success,  and  where  men  and  equip¬ 
ment  are  involved,  of  increased  chances  for  survivability. 

How  does  the  target  recognition  capability  that  is  needed 
for  weapon  delivery  relate  to  the  more  complex  evaluation  of 


reconnaissance  images?  It  appears  that  the  recognition  algo¬ 
rithms  are  a  subset  of  the  reconnaissance  set.  Target  extrac¬ 
tion  is  a  key  element  in  reconnaissance,  but  so  is  the  extrac¬ 
tion  of  information  which  is  only  indirectly  related  to  targets. 
As  the  problems  shift  from  tactical  to  strategic  in  nature, 
their  subtlety  increases  further.  Present  algorithms  are  satis¬ 
factory  for  only  the  simplest  of  tactical  problems.  Improve¬ 
ments  in  performance  for  any  application  will  require  increasing¬ 
ly  complex  algorithms  that  make  use  of  all  relevant  image 
information.  This  is  the  challenge  to  the  algorithm  designers. 

It  is  an  exercise  in  pure  research  in  image  understanding,  but 
with  the  prospect  that  positive  results  will  be  applied  to  real 
problems  with  very  little  delay. 

Following  the  initiation  and  feasibility  testing  of  an  algo¬ 
rithm  concept,  extensive  statistical  testing  is  desired,  on 
realistic  data  bases,  in  order  to  establish  performance  measures. 
In  view  of  the  complexity  of  the  algorithms  which  are  currently 
under  evaluation,  the  statistical  test  programs  require  special 
processing  considerations  in  order  to  minimize  execution  times 
and  computational  cost. 

In  addition  to  the  considerations  noted  above,  the  weapon 
delivery  scenario  brings  with  it  special  demands,  including  the 
requirement  for  real-time  performance  with  equipment  having 
severe  limitations  in  size,  weight,  power,  and  cost.  The  algo¬ 
rithm  designer  must  be  aware  of,  and  responsive  to,  the  feasi¬ 
bility  of  the  implementation  of  his  designs  in  real-time  hardware. 


3 . 2  The  approach 
A.  Maryland 


Maryland's  approach  to  image  understanding  and  analysis 
emphasizes  highly  parallel,  cooperating  processes  which  are 
especially  suitable  for  real-time  implementation,  particularly 
a  few  years  from  now  when  parallel  cellular  hardware  becomes 
readily  available,  so  that  it  becomes  possible  to  assign  a 
processor  to  each  image  pixel.  Since  such  processes  make  use 
of  very  little  knowledge  at  the  pixel  level,  they  are  subject 
to  errors ,  which  would  normally  imply  the  need  for  backtracking 
at  later  stages  of  the  analysis;  this  would  be  both  difficult 
and  time  consuming.  To  avoid  this  problem,  the  initial  indi¬ 
vidual  decisions  are  made  fuzzily  or  probabilistically,  and 
are  then  checked  for  consistency  with  other  decisions  by  an 
iterative  "relaxation"  process.  This  strategy  of  deferred  com¬ 
mitment  and  use  of  convergent  evidence  is  designed  to  minimize 
the  backtracking  problem. 

Another  way  in  which  convergent  evidence  can  be  employed 
at  the  early  stages  of  image  analysis  is  through  the  joint  use 
of  operators  of  several  types,  possibly  having  a  range  of  sizes. 
For  example,  gray  level  clustering  (i.e.,  thresholding)  and 
edge  detection  are  used  jointly  in  the  SUPERSLICE  and  SUPERLINK 
algorithms.  As  another  example,  spot  detectors  of  several 
sizes  can  be  used  to  detect  bloblike  objects  in  an  image,  and 
local  thresholds  can  then  be  selected,  based  on  the  detector 
responses,  to  extract  the  objects.  This  last  example  illustrates 


how  simple  types  of  size  and  shape  information  about  the 
objects  that  it  is  desired  to  extract  can  be  used  to  influence 
the  extraction  process  so  as  to  favor  their  extraction. 

Texture  analysis  too  provides  some  good  possibilities  for 
the  use  of  cooperative,  parallel  computation.  Conventional 
texture  features  are  simple  gray  level  or  local  property 
statistics  computed  over  small  windows  of  an  image.  A  poten¬ 
tially  more  powerful  approach  to  understanding  texture  is  to 
decompose  the  given  texture  into  "primitive  elements",  and 
measure  properties  of  these  primitives,  as  proposed  by  Maleson 
et  al.  (As  a  compromise,  one  can  measure  gray  level  or  local 
property  statistics  at  selected  points  which  are  expected  to 
be  in  given  positions  relative  to  the  primitives,  e.g.  on  or 
near  their  borders;  this  is  the  "generalized  cooccurrence" 
aPProach  of  Davis  et  al.)  Parallel  methods  of  texture  primitive 
extraction,  by  clustering  pairs  of  antiparallel  edges,  yield 
excellent  sets  of  primitives.  Cooperative  methods  can  also 
be  used  to  increase  the  reliability  of  texture  features  measured 
over  small  image  windows;  this  makes  it  possible  to  use  smaller 
windows  and  thus  reduce  the  occurrence  of  mixed  windows  in  an 
image.  These  approaches  are  being  extended  to  multiple  resolu¬ 
tions  and  multiple  window  sizes. 

Edge  clustering  is  also  an  economical  approach  to  the 
detection  of  cultural  features  such  as  roads  and  buildings  in 


imagery.  An  approach  is  under  investigation  in  which  edge 
information  is  iteratively  enhanced,  by  a  relaxation- like 
process ,  and  edge  pixels  are  then  linked  into  edge  segments. 
These  segments  are  then  clustered,  in  a  Hough-like  space,  on 
the  basis  of  collinearity  and  antiparallelness,  yielding  pieces 
of  linear  features.  Finally,  a  relaxation  approach  is  used  to 
probabilistically  label  these  pieces  as  to  feature  type,  and 
iteratively  adjust  the  probabilities  so  as  to  reconcile  the 
labels  of  related  pieces,  thus  obtaining  a  consistent  global 
labeling. 

The  relaxation  approach  is  also  being  used  for  shape  recog¬ 
nition  and  matching.  The  border  of  a  given  shape,  or  collection 
of  touching  shapes,  is  ambiguously  segmented,  and  the  segments 

probabilistically  labelled.  The  probabilities  are  then  ad¬ 
justed  based  on  their  compatibilities  with  those  of  related 
segments.  This  yields  a  relatively  unambiguous  labeling  from 
which  it  is  easy  to  find  segment  sequences  that  correspond  to 
shapes  of  the  desired  types.  This  approach  is  being  extended 
to  handle  hierarchical  shape  descriptions.  Cooperative  tech¬ 
niques  can  also  be  used  at  a  lower  level  to  produce  the  original 
shape  segmentations. 

Compact  hierarchical  representations  for  images  and  regions , 
based  on  quadtree  data  structures,  have  been  extensively  studied 
on  t-he  current  project.  It  is  planned  to  extend  this  work 
from  exact  to  approximate  representation;  a  block  is  subdivided 


if  its  contents  are  sufficiently  nonuniform.  (This  is,  of 
course,  the  well-known  approach  to  image  segmentation  by  recur¬ 
sive  splitting. )  Algorithms  will  be  developed  for  efficiently 
computing  properties  of  images  directly  from  this  representa¬ 
tion,  and  for  converting  between  this  and  other  representations. 
These  will  generalize  the  algorithms  developed  for  the  case  of 
exact  representation  on  the  current  project,  and  will  make  the 
work  applicable  to  a  wide  class  of  images. 

The  principal  goal  of  the  project  will  be  the  study  of 
algorithms  that  can  be  used  for  object  detection  on  tactical 
imagery,  and  that  are  potentially  implementable  in  real  time. 
There  will  be  close  collaboration  with  the  U.S.  Army  Night 
Vision  and  Electro— Optics  Laboratory,  as  monitoring  agency, 
as  well  as  with  Westinghouse ,  in  selecting  appropriate  problems 
from  this  domain. 

Many  of  the  algorithms  developed  on  this  project  will  also  be 
applicable  to  the  planned  DARPA/DMA  Image  Understanding  testbed. 
It  is  proposed  to  contribute  a  collection  of  such  algorithms 
for  the  testbed  in  the  form  of  an  interactive  program  that  will 
allow  the  user  to  define  his  problem,  prompt  him  to  provide 
traning  samples  (e.g. ,  of  objects  and  background,  of  textures, 
etc.)  for  analysis,  recommend  applicable  techniques,  apply  them 
to  the  image,  and  display  the  results  for  evaluation.  This  will 
provide  an  extensible  capability  for  interactive  selection  and 
evaluation  of  segmentation  and  texture  analysis  techniques  as 


modular  front  ends  to  an  image  analysis  process.  This  capa¬ 
bility  should  be  useful  in  connection  with  most  of  the  imagery 
that  other  contributors  to  the  testbed  will  analyze,  or  that 
users  of  the  testbed  may  want  to  analyze. 

B.  Westinghouse 

During  the  current  DARPA  program  Westinghouse  has  explored 
the  feasibility  of  the  LSI  implementation  of  a  series  of  algo¬ 
rithms  developed  by  Maryland  for  higher  level  image  processing, 
particularly  in  the  area  of  relaxation  processes.  Concurrent 
with  this  effort  a  second  potential  task  of  algorithm  evalua¬ 
tion  has  emerged.  This  is  because  the  Maryland  algorithms 
are  sufficiently  complex  as  to  require  many  minutes  to  process 
a  single  sample  window  on  a  general-purpose  computer.  Since  a 
statistical  test,  as  developed  by  NVEOL  for  example,  may 
include  400  to  1000  image  samples,  detailed  consideration  must 
be  given  to  the  test  approach.  A  solution  to  this  problem  has 
been  proposed  at  Westinghouse,  using  fully  programmable  array 
processors  as  a  means  to  achieve  high  throughput  with  great 
flexibility.  It  offers  the  additional  advantage  of  a  means 

LSI  implementation  with  little  or  no  reprogramming,  since 
Westinghouse  is  developing  a  series  of  LSI  chips  (Universal 
Arrays)  for  implementation  of  array  processors. 

The  proposed  program  of  algorithm  evaluation  and  implemen¬ 
tation  has  emerged  in  the  form  indicated  by  Figure  13.  This 
diagram  portrays  the  evolution  of  a  miniaturized  image  proces- 

fr°m  the  initial  concept  to  hardware .  The  steps  performed 
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Figure  13.  EVOLUTION  FROM  PROCESSING  CONCEPT  TO  MINIATURIZED  HARDWARE 


by  the  University  of  Maryland  are  shown  across  the  top  of  the 
figure.  Following  algorithm  development,  Maryland  will  perform 
feasibility  testing  with  a  limited  data  base,  using  a  general- 
purpose  computer  for  simulation.  With  the  test  results  in 
hand,  as  well  as  the  algorithm  statement  itself,  Westinghouse 
will  develop  the  necessary  programming  to  perform  a  statistical 
test  of  the  algorithm  with  a  data  base  selected  or  approved 
by  NVEOL,  and  involving  tactical  imagery.  Westinghouse  may  also 
assist  in  data  base  compilation  if  necessary,  using  its  labora¬ 
tories  for  conversion  of  video  data  into  digital  format. 

In  addition  to  examining  algorithms  generated  by  Maryland, 
Westinghouse  proposes  to  examine  algorithms  described  by  other 
university  participants  in  the  Image  Understanding  Program. 
Algorithms  of  interest  will  be  statistically  tested  on  the 
same  data  bases.  Test  results  will  be  provided  to  all  program 
participants . 

The  algorithm  design,  together  with  the  statistical  test 
results,  will  form  the  basis  for  a  design  feasibility  study 
for  implementation  of  miniaturized  hardware.  This  effort  will 
be  aided  by  company-sponsored  investigations  of  the  available 
LSI  families,  and  their  associated  system  architectures.  The 
results  will  be  made  available  to  various  military  agencies, 
such  as  NVEOL,  for  further  design  and  construction  effort. 


3.3  Statement  of  Work 


A.  Maryland 

The  proposed  project  will  emphasize  the  development  of 
image  analysis  algorithms  suitable  for  real-time  implementation. 
Specific  areas  of  investigation  will  include: 

a.  Segmentation  techniques  for  object  extraction  that 
make  use  of  convergent  evidence--e . g . ,  gray  level 
clustering  in  conjunction  with  edge,  spot,  or  streak 
detection,  at  multiple  resolutions. 

b.  Texture  analysis  techniques  for  background  analysis 
and  terrain  classification,  based  on  the  extraction  and 
characterization  of  primitive  texture  elements,  as  well 
as  on  the  use  of  multiple-resolution  interactions  to 
enhance  texture  property  measurements. 

c.  Methods  of  extracting  macroscopic  image  features  such 
as  roads  and  buildings  by  clustering  of  edges  into 
feature  segments,  and  using  relaxation— like  techniques 
to  find  consistent  labellings  of  these  segments. 

d.  Shape  recognition  techniques  based  on  ambiguous  seg¬ 
mentation  and  on  the  use  of  relaxation  methods  to  find 
consistent  sets  of  segments.  Relaxation  will  also  be 
applied  to  pattern  detection  problems,  e.g.,  to  detect¬ 
ing  arrays  of  jointly  occurring  small  objects  which 
satisfy  given  constraints  on  their  relative  positions. 

e.  The  use  of  compact  representations  such  as  quadtrees 
for  image  approximation  and  compression,  as  well  as  for 


the  exact  representation  of  regions  so  as  to  provide 
efficient  methods  of  handling  region  databases. 

In  conjunction  with  NVEOL  and  Westinghouse,  a  set  of 
practical  analysis  tasks  will  be  selected,  together 
with  appropriate  databases.  The  "Alabama"  data  base, 
consisting  of  FLIR  images  of  tactical  targets  at  low 
and  high  resolution,  will  be  used  initially,  since 
test  results  with  it  have  been  obtained  for  several 
existing  algorithms.  The  "Ft.  Polk"  and  "A.P.  Hill" 
data  bases  increase  the  variety  of  tactical  targets  that 
can  be  observed,  as  well  as  the  background  conditions. 
Target  motion  and  obscuration  are  also  present.  Addi- 
tional  databases  available  at  Westinghouse  can  further 
increase  the  variety  of  sensor  types,  target  character¬ 
istics,  and  environmental  conditions. 

It  is  desired  by  NVEOL  to  acquire  additional  data¬ 
bases  which  provide  contextual  information  to  assist  in 
target  classification,  including  the  occurrence  of  mili¬ 
tary  target  groups  or  clusters.  Use  of  such  contextual 
information  is  an  important  aspect  of  higher  level 
image  understanding. 

Algorithms  will  be  tested  on  small  data  sets,  and 
promising  algorithms  will  be  recommended  to  Westinghouse 
for  further  testing  and  for  possible  hardware  design. 


g.  It  is  also  planned  to  provide  a  collection  of  algorithms, 
particularly  in  areas  (a-c) ,  for  use  in  the  DARPA/DMA 
testbed,  in  the  form  of  an  interactive  package.  As 
presently  envisioned,  the  package  will  provide  users 
with  a  menu  of  basic  segmentation,  texture  analysis,  and 
feature  extraction  techniques.  Users  will  not  need  to 
be  familiar  with  these  techniques;  they  will  be  prompted 
to  provide  information  on  the  basis  of  which  the  system 
will  make  choices  and  present  results  for  evaluation. 

For  example,  if  a  user  wishes  to  extract  objects  from  a 
background,  he  will  be  asked  to  define  (via  cursor) 
samples  of  the  objects  and  background;  the  system  will 
then  select  an  appropriate  segmentation  technique, 
apply  it  to  the  image,  and  request  him  to  critique  the 
results — e.g.,  to  indicate  errors  (via  cursor) — so  that 
the  technique  can  be  refined  as  necessary.  It  is  planned 
to  work  closely  with  SRI,  as  testbed  implementors,  so 
that  this  package  can  be  closely  integrated  with  the 
testbed  software. 

B.  Westinghouse 

The  proposed  program  will  be  directed  toward  the  evalua¬ 
tion  and  implementation  of  image  processing  algorithms  developed 
by  the  University  of  Maryland  or  others  involved  in  the  DARPA 


Image  Understanding  program.  Initial  effort  will  be  concerned 
with  the  inclusion  of  relaxation  operations  in  the  preDrocessor 
functions.  Results  will  be  compared  with  the  Maryland  "Super¬ 
slice"  and  Westinghouse  AUTO-Q  algorithms.  In  addition,  close 
attention  will  be  given  to  the  "primal  sketch"  operations 
described  by  MIT. 

Specific  tasks  will  include: 

a.  Review  of  Maryland  and  other  IU  programs  for  candidate 
algorithms . 

b.  Acquisition  and  refinement  of  data  bases  appropriate 
to  both  weapon  delivery  and  reconnaissance  scenarios, 
and  determination  of  candidate  scenarios  in  coordina¬ 
tion  with  NVEOL.  Such  scenarios  include  the  helicoptor 
pop-up  operation,  and  target  handoff  from  an  aircraft 
sensor  to  a  missile  seeker. 

c.  Statistical  testing  of  algorithms  using  appropriate 
simulation  tools,  such  as  array  processors.  Results 
to  be  reported  to  algorithm  designer. 

Perform  design  feasibility  analyses  for  hardware  imple¬ 
mentation  of  successful  algorithms,  with  consideration 
of  available  LSI  families.  Provide  recommendations 
for  applications  of  successful  algorithms  to  military 
systems . 

e.  Deliverable  items  under  the  contract  will  be  the  test 
results  on  candidate  algorithms  and  hardware  design 
information,  including  resources  required  for  imple- 


mentation.  Reports  will  contain  comparative  evalua¬ 
tions  of  algorithms.  Transportable  software  will  be 
provided. 

Success  on  the  program  will  be  measured  by  the  performance 
improvements  which  are  statistically  demonstrated  in  terms  of 
target  detection  and  classification  rates,  and  the  reduction 
of  false  alarms.  Such  improvements  will  constitute  advances 
in  the  state  of  the  art,  and  will  thus  assure  that  hardware 
implementation  will  occur.  This  sequence  of  events  was  demon¬ 
strated  during  the  first  phase  of  this  program,  with  the  "Super- 
slice"  algorithm. 


4.  Publications 


This  section  lists  status  reports,  technical  reports,  and 
papers  published  on  the  current  phase  of  the  project,  under 
the  following  headings: 

A.  Semiannual  reports  (as  well  as  the  Final  Report  on  the 
previous  phase  of  the  project),  project  status  reports, 
and  quarterly  reports  on  the  Westinghouse  subcontract 

B.  Technical  reports 

C.  Papers 

For  convenience,  a  brief  outline  of  the  individual  research 
areas  on  the  project  is  given  in  Sections  4.1-2,  with  references 
to  the  reports  that  relate  to  each  area. 


4 . 1  Object  detection 


Topic 


a.  Preprocessing 


b.  Edge  detection 


c.  Corner,  line, 
and  strip 
detection 


d.  Pixel  clas¬ 
sification 


e.  Segmentation 


Noise  cleaning  techniques 
"Probability  transforms" 
Gaussian  convolution 


Report  Nos . 

29,29,34,71,72 

38 

63 


Color  edge  detection  18 

Straight  edge  enhancement  22 

Step-fitting  edge  detection  33 

Pyramid  edge  enhancement  64 

Edge  maxima  6  7 

Strip  detection  27 

Antiparallel  linking  57 

Collinearity  linking  67 

Gray-level  corner  detection  69 

Clustering  of  collinear 

segments  70 

ISODATA  thresholding  35 

Relaxation  thresholding  55 

Relaxation  "busyness" 

clustering  65 

Pyramid  thresholding  68 

SUPERSLICE  19 

SUPERLINK  24 

Blob  detection  by  relaxation  50 

Pyramid  blob  and  streak 

detection  52,60 

Pyramid  "SUPERSLICE"  66 


4.2  Tools  for  image 


understanding 

Topic 

Report  Nos. 

a. 

Image  models 

Mosaic  model  fitting 

32,47 

b . 

Texture  analysis 

Generalized  cooccurrence 

30 

Primitive  extraction 

40 

Feature  smoothing 

56 

c . 

Shape  analysis 

Segmentation  by  relaxation 

39,48 

Segmentation  by  global 
features 

59 

d. 

Region  repre¬ 
sentation 

Quadtrees  and  pyramids 

28,31,36,37 
41-46 ,49,51 
54,58,62 

Medial  axes 

61,73 

e . 

Matching 

Point  patterns 

21,23,26 

Relational  structures 

17,25 

f . 

Software 

Control  structures 

16 

Utility  packages 

53,55 

A.  Project  reports 


A1 .  Final  Report  on  first  phase 

1.  Algorithms  and  Hardware  Technology  for  Image  Recognition, 

Final  Report,  March  31,  1978. 

A2 .  Semiannual  reports  and  project  status  reports;  Image 
Understanding  Using  Overlays 

2.  Semiannual  report,  1  April-30  September  1978. 

3.  Semiannual  report,  1  October  1978-31  March  1979. 

4.  Semiannual  report,  1  April-30  September  1979. 

5.  Project  status  report,  1  April-30  September  1978,  in 
Proceedings,  Image  Understanding  Workshop,  November  1978, 

20-27. 

6.  Project  status  report,  1  October-31  March  1979,  in  Proceedings, 
Image  Understanding  Workshop,  April  1979,  14-24. 

7.  Project  status  report,  1  April-30  September  1979,  in 
Proceedings,  Image  Understanding  Workshop,  November  1979, 
166-175. 

8.  Project  status  report,  1  October  1979-31  March  1980,  in 
Proceedings,  Image  Understanding  Workshop,  April  1980,1-12. 

■^3.  Quarterly  reports  on  Westinghouse  subcontract:  Architecture 
for  Higher  Level  Digital  Image  Processing 

9.  First  quarterly  report,  July  30,  1978. 

10.  Second  quarterly  report,  October  30,  1978. 

11.  Third  quarterly  report,  January  30,  1979. 

12.  Fourth  quarterly  report,  April  30,  1979. 

13.  Fifth  quarterly  report,  July  31,  1979. 

14.  Sixth  quarterly  report,  October  31,  1979. 

Seventh  quarterly  report,  January  31,  1980. 


15. 


B .  Technical  reports 


16.  Martin  Herman,  "A  System  for  Control  Structure  Implemen¬ 
tation  for  Image  Understanding."  TR-646,  March  1978. 

17.  Les  Kitchen,  "Discrete  Relaxation  for  Matching  Relational 
Structures."  TR-665,  June  1978. 


18.  P.  V.  Sankar ,  "Color  Edge  Detection:  A  Comparative  Study." 
TR-666 ,  June  1978. 

19.  D.  L.  Milgram,  "Region  Extraction  Using  Convergent 
Evidence."  TR-674,  June  1878. 

20.  Judith  P.  Davenport,  "A  Comparison  of  Noise  Cleaning 
Techniques."  TR-689*  September  1978. 

21.  Daryl  J.  Kahl,  Azriel  Rosenfeld,  and  Alan  Danker,  "Some 
Experiments  in  Point  Pattern  Matching."  TR-690,  September 

1978. 

22.  Shmuel  Peleg  and  Azriel  Rosenfeld,  "Straight  Edge  Enhancement 
and  Mapping."  TR-694,  September  1978. 

23.  San jay  Ranade  and  Azriel  Rosenfeld,  "Point  Pattern  Matching 
by  Relaxation."  TR-702,  October  1978. 

24.  David  L.  Milgram,  "Edge  Point  Linking  Using  Convergent 
Evidence."  TR-704,  October  1978. 

25.  Les  Kitchen,  "Relaxation  Applied  To  Matching  Quantitative 
Relational  Structures."  TR-707,  October  1978. 

26.  Daryl  J.  Kahl,  "Sketch  Matching."  TR-716,  November  1978. 

27.  Alan  Danker  and  Azriel  Rosenfeld,  "Strip  Detection  Using 
Relaxation."  TR-725,  January  1979. 

28.  Charles  R.  Dyer,  Azriel  Rosenfeld,  and  Hanan  Samet,  "Region 
Representation:  Boundary  Codes  from  Quadtrees."  TR-732, 
February  1979. 

29.  Ann  Scher,  Flavio  R.  D.  Velasco,  and  Azriel  Rosenfeld, 

"Some  New  Image  Smoothing  Techniques."  TR-733,  Februarv 

1979. 

30.  Charles  R.  Dyer,  Tsai -Hong  Hong,  and  Azriel  Rosenfeld, 

Texture  Classification  Using  Gray  Level  Cooccurrence 
Based  on  Edge  Maxima."  TR-738,  March  1979. 


Hanan  Samet,  "Region  Representation:  Quadtrees  from 
Boundary  Codes."  TR-741,  March  1979. 

Tsvi  Dubitzki,  Narendra  Ahuja,  and  Azriel  Rosenfeld, 
"Predicted  Properties  of  Mosaic  Images."  TR-746,  March 


Azriel  Rosenfeld,  "The  Simplest  'Hueckel'  Edge  Detector  is 
a  Roberts  Operator."  TR-747,  March  1979. 

Judith  P.  Davenport  and  Azriel  Rosenfeld,  "A  Comparison 
of  Noise  Cleaning  Techniques  for  Color  Images . "  TR-748 

April  1979. 

Flavio  R.  D.  Velasco,  "Thresholding  Using  the  Isodata 
Clustering  Algorithm."  TR-751,  March  1979. 

Hanan  Samet,  "Computing  Perimeters  of  Images  Represented 
by  Quadtrees."  TR-755,  April  1979. 

Hanan  Samet,  "Connected  Component  Labeling  Using  Quadtrees. 
TR-756,  April  1979. 

Ann  Scher  and  Azriel  Rosenfeld,  "'Probability  Transforms' 
of  Digital  Pictures."  TR-758,  April  1979. 

Wallace  S.  Rutkkowski,  Shmuel  Peleg,  and  Azriel  Rosenfeld, 
"Shape  Segmentation  Using  Relaxation."  TR-762,  May  1979. 

Tsai-Hong  Hong,  Charles  R.  Dyer,  and  Azriel  Rosenfeld. 
"Texture  Primitive  Extraction  Using  an  Edge-Based  Approach 
TR-763 ,  May  1979. 

Hanan  Samet,  "Region  Representation:  Raster-to-Quadtree 
Conversion."  TR-766,  May  1979. 

Hanan  Samet,  "Region  Representation:  Quadtrees  from  Binary 
Arrays."  TR-767,  May  1979. 

Hanan  Samet,  "Region  Representation:  Quadtree-to-Raster 
Conversion."  TR-768,  June  1979. 

Charles  R.  Dyer,  "Computing  the  Euler  Number  of  an  Image 
from  Its  Quadtree."  TR-769,  May  1979. 

Michael  Shneier ,  "Linear  Time  Calculations  of  Geometric 
Properties  Using  Quadtrees."  TR-770,  May  1979. 


46.  Hanan  Samet,  A  Distance  Transform  for  Images  Represented 
by  Quadtrees."  TR-780,  July  1979. 

47.  Narendra  Ahuja  and  Azriel  Rosenfeld,  "Fitting  Mosaic  Models 
to  Textures."  TR-789,  July  1979. 

48.  Wallace  S.  Rutkowski,  "Shape  Segmentation  Using  Relaxation, 
II."  TR-793 ,  July  1979. 

49.  Michael  Shneier,  "A  Path-Length  Distance  Transform  for 
Quadtrees."  TR-794,  July  1979. 

50.  Alan  Danker,  "Blob  Detection  by  Relaxation."  TR-795 
July  1979. 

51.  Hanan  Samet,  "A  Quadtree  Medial  Axis  Transform."  TR-803 
August  1979. 

52.  Michael  Shneier,  "Using  Pyramids  to  Define  Local  Thresholds 
for  Blob  Detection."  TR-808,  September  1979. 

53.  Robert  L.  Kirby,  Russ  Smith,  Philip  A.  Dondes ,  San jay 
Ranade,  Les  Kitchen,  and  Fred  Blonder,  "Interfaces,  Sub¬ 
routines,  and  Programs  for  the  Grinnell  GMR-27  Display 
Processor  on  a  PDP-11/45  with  the  UNIX  Operating  System. " 
TR-810,  October  1979. 

54.  Peter  J.  Burt,  "Tree  and  Pyramid  Structures  for  Coding 
Hexagonally  Sampled  Binary  Images."  TR-814,  October  1979. 

55.  Russell  C.  Smith,  "A  General-Purpose  Software  Package  for 
Array  Relaxation."  TR-839,  December  1979. 

56.  Tsai-Hong  Hong,  Angela  Y.  Wu,  and  Azriel  Rosenfeld,  "Feature 
Value  Smoothing  as  an  Aid  in  Texture  Analysis."  TR— 844 
December  1979. 

57.  Ann  Scher,  Michael  Shneier,  and  Azriel  Rosenfeld,  "A  Method 
for  Finding  Pairs  of  Anti-parallel  Straight  Lines."  TR-845, 
December  1979. 

58.  San j  ay  Ranade ,  Azriel  Rosenfeld,  and  Hanan  Samet,  "Shape 
Approximation  Using  Quadtrees."  TR-847,  December  1979. 

59.  W.  S.  Rutkowski,  "Shape  Segmentation  Using  Arc/Chord 
Properties."  TR-849,  December  1979. 

60.  Michael  Shneier,  "Extracting  Linear  Features  from  Images 
Using  Pyramids."  TR-855,  January  1980. 

61.  Shmuel  Peleg  and  Azriel  Rosenfeld,  "A  Min-Max  Medial  Axis 
Transformation."  TR-856,  January  1980. 


62.  Hanan  Samet,  "Neighbor  Finding  Techniques  for  Images 
Represented  by  Quadtrees."  TR-857,  January  1980. 

63.  Peter  J.  Burt,  "Fast,  Hierarchical  Correlations  with 
Gaussian-Like  Kernels."  TR-860,  January  1980. 

64.  Sanjay  Ranade,  "Use  of  Quadtrees  for  Edge  Enhancement." 

TR-862 ,  February  1980. 

65.  Philip  A.  Dondes  and  Azriel  Rosenfeld,  "Pixel  Classifica¬ 
tion  Based  on  Gray  Level  and  Local  'Busyness.'"  TR-874 
March  1980. 

66.  Sanjay  Ranade,  Azriel  Rosenfeld,  and  Judith  M.  S.  Prewitt, 
"Use  of  Quadtrees  for  Image  Segmentation."  TR-878, 

February  1980. 

67.  Les  Kitchen,  Alan  Broder,  and  Azriel  Rosenfeld,  "Two  Notes 
on  Digital  Edges  and  Lines."  TR-885,  March  1980. 

68.  Angela  Y.  Wu,  Tsai-Hong  Hong,  and  Azriel  Rosenfeld, 

"Threshold  Selection  Using  Quadtrees."  TR-886,  March  1980. 

69.  Les  Kitchen  and  Azriel  Rosenfeld,  "Gray-level  Corner 
Detection."  TR-887,  April  1980. 

70.  Ann  Scher,  Michael  Shneier,  and  Azriel  Rosenfeld,  "Cluster¬ 
ing  of  Collinear  Line  Segments."  TR-888,  April  1980. 

71.  Sanjay  Ranade  and  Michael  Shneier,  "Using  Quadtrees  to 
Smooth  Images."  TR-894,  April  1980. 

72.  Shmuel  Peleg,  "Elimination  of  Seams  from  Photomosaics." 

TR-895 ,  April  1980. 

73.  Shyuan  Wang,  Angela  Y.  Wu,  and  Azriel  Rosenfeld,  "Image 
Approximation  from  Grayscale  'Medial  Axes.'"  TR-900,  May  1980. 


C .  Papers 


Cl .  DARPA  Image  Understanding  Workshops 

74.  D.  L.  Milgram,  Edge  point  linking  using  convergent 
evidence  (November  1978,  85-91). 

75.  A.  Rosenfeld,  Some  experiments  in  matching  using  relaxa¬ 
tion  (November  1978,  110-114). 

76.  T.  J.  Willett,  Hardware  implementation  of  image  processing 
using  overlays:  relaxation  (November  1978,  175-181). 

77.  A.  Rosenfeld,  A.  Danker,  and  C.  R.  Dyer,  Blob  extraction 
by  relaxation  (April  1979,  61-65). 

78.  T.  J.  Willett,  C.  W.  Brooks,  and  G.  E.  Tisdale,  Relaxation, 
systolic  arrays,  and  universal  arrays  (April  1979,  164-170). 

79 •  T.  J.  Willett,  A.  R.  Helland,  and  G.  E.  Tisdale,  Higher 
level  algorithms:  evaluation  and  implementation  (November 
1979,  15-24). 

80.  C.  Rieger,  ZMOB:  A  mob  of  256  cooperative  Z80A-based  micro¬ 
computers  (November  1979,  25-30). 

81.  M.  Shneier,  Using  pyramids  to  define  local  thresholds  for 
blob  detection  (November  1979,  31-35). 

82.  H.  Samet  and  A.  Rosenfeld,  Quadtree  structures  for  region 
processing  (November  1979,  36-41). 

83.  A.  Rosenfeld,  Cooperative  computation  in  texture  analysis 
(November  1979,  52-56) . 

84.  A.  Rosenfeld,  Levels  of  representation  in  cultural  feature 
extraction  (November  1979,  112-127). 

85.  M.  Tavakoli,  Toward  the  recognition  of  cultural  features 
(April  1980,  33-57) . 

86.  A  Rosenfeld,  Some  uses  of  pyramids  in  image  processing  and 
segmentation  (April  1980,  112-120). 

87.  A.  R.  Helland,  Convergence  properties  of  two  label  relaxa¬ 
tions  (April  1980,  176-181). 


c2*  Journal  and  conference  papers  (accepted  or  published) 

88.  L.  Kitchen  and  A.  Rosenfeld,  Discrete  relaxation  for 
matching  relational  structures,  IEEETSMC-9 ,  1979,  869-874. 

89.  D.  L.  Milgram,  Region  extraction  using  convergent  evidence, 
CGIP  11,  1972,  1-12. 

90.  D.  J.  Kahl ,  A.  Rosenfeld,  and  A.  Danker,  Some  experiments 
in  point  pattern  matching,  IEEETSMC-10 ,  1980,  105-116. 

91.  S.  Ranade  and  A.  Rosenfeld,  Point  pattern  matching  by 
relaxation,  Pat.  Recog.  12,  1980,  in  press. 

92.  L.  Kitchen,  Relaxation  applied  to  matching  quantitative 
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Appendix.  Summaries  of  Quarterly  Reports  on  Westinghouse 
Subcontract:  "Architecture  for  Higher  Level 
Digital  Image  Processing" 

1.  First  Quarterly  Report,  July  30,  1978. 

The  report  begins  with  a  review  of  desired  system  design 
goals.  This  is  followed  by  a  description  of  available  micropro¬ 
cessor  hardware,  a  review  of  the  LISP  approach  to  the  manipulation 
of  list  structures,  and  a  preliminary  discussion  of  the  processing 
required  to  implement  relaxation  methods  of  object  classification. 
The  report  concludes  with  a  description  of  specific  bit-slice 
processors . 

2.  Second  Quarterly  Report,  October  30,  1978. 

This  report  begins  with  a  continuation  of  a  description  of 
bit  slice  microprocessors  with  the  emphasis  on  control  units 
this  time,  and  an  examination  of  commercially  available  units. 
Several  more  Maryland  algorithms,  namely  non-linear  probabilistic 
relaxation,  connected  components,  and  Superlink  are  described. 
Hardware  implementations  for  non-linear  probablistic  relaxation 
and  connected  components  are  described  in  the  next  section.  The 
final  section  shows  some  tentative  conclusions,  in  light  of  the 
above  continuing  analysis,  regarding  an  appropriate  architecture 
for  image  processing  both  for  the  image  processing  module  and 
the  array  of  modules. 


3.  Third  Quarterly  Report,  January  30,  1979. 

This  report  begins  with  a  description  of  the  computations 
for  performing  relaxation  at  the  pixel  level  in  order  to  divide 
an  image  into  dark  and  light  regions.  The  computations  are  re¬ 
structured  to  conform  to  a  matrix  multiplied  by  a  vector,  mul¬ 
tiplied  by  a  scalar.  A  "Systolic"*  array  of  processors  is 
applied  to  this  computational  structure  and  the  results  described. 
The  individual  processors  take  a  particular  form  of  the  Westing- 
house  Universal  Array  currently  being  developed  for  20  mega¬ 
operations/sec.  signal  processors.  One  personalization  of  the 
Universal  Array  in  the  systolic  array  architecture  is  a  self- 
contained,  4x4  multiply  chip  with  a  12-nanosecond  8-bit  product. 
The  computations  are  also  restructured  to  delete  frame  storage 
and  permit  computational  speeds  approaching  real  time,  while 
still  maintaining  a  small  volume. 

4.  Fourth  Quarterly  Report,  April  30,  1979. 

This  report  considers  hardware  implementation  of  Relaxation 
Processes  using  cellular  automata  architecture  with  Universal 
Arrays  as  the  central  solid  state  module.  The  work  examines  the 
edge/no  edge  and  light/dark  algorithms.  In  particular,  an 
effort  has  been  directed  to  the  development  of  a  cellular  array 


*Kung,  H.  T.,  and  Leiserson,  C.  E.,  Systolic  Arrays  for  VLSI, 
Dept,  of  Computer  Science,  Carnegie-Mellon  University,  Dec., 
1978.  "Systolic  Array"  is  a  term  used  by  H.  T.  Rung  to  describe 
a  network  of  processors  which  rhythmically  compute  and  pass 
data  through  the  system. 


module  which  shows  promise.  It  can  be  operated  at  speeds 
comparable  to  frame  rates,  and  in  part  can  be  composed  of  a 
Universal  Array  multiplier  and  a  Universal  Array  multiplexer. 

5.  Fifth  Quarterly  Report,  July  31,  1979. 

This  report  deals  primarily  with  the  definition  of  digital 
architecture  for  implementing  the  image  processing  algorithms 
developed  at  the  University  of  Maryland.  A  need  has  also 
emerged  to  provide  support  to  the  University  of  Maryland  in  the 
statistical  testing  of  complex  algorithms.  This  had  led  to  an 
investigation  of  programmability  of  these  algorithms  on  a  fully 
programmable  array  processor  (PAP)  developed  at  Westinghouse. 

6.  Sixth  Quarterly  Report,  October  31,  1979. 

This  report  covers  results  of  special  analysis  performed  as 
part  of  the  recent  work  to  support  the  University  of  Maryland  in 
the  statistical  testing  of  complex  algorithms.  The  planned 
steps  in  this  support  program  are: 

Selection  of  processing  algorithms  for  evaluation. 
Analysis  and  adaptation  of  algorithms  for  execution  on 
the  Programmable  Array  Processor  (PAP) . 

Evaluation  of  algorithms  on  a  PDP-VAX  GP  computer. 
Throughput  analysis;  PAP  vs.  VAX. 

Processing  of  a  set  of  imagery. 

Results  are  reported  at  regular  intervals  as  appropriate. 


This  report  analyzes  the  segmentation  properties  of  gray 
level  relaxation  applied  to  the  two-label  case.  The  results 
produce  threshold,  speed  and  stability  criteria  to  facilitate 
subsequent  processing.  Most  of  these  results  are  experimentally 
verified  in  a  report  by  Azriel  Rosenfeld  and  Russell  C.  Smith. 
Evaluation  and  verification  is  in  progress  at  Westinghouse 
using  comparable  imagery  and  test  patterns.  The  current  status 
of  the  computer  program  modeling  and  test  results  will  be  covered 
in  a  separate  report  to  be  completed  shortly.  Efforts  underway 
and  planned  for  the  immediate  future  include  evaluation  of  image 
samples  on  the  VAX/,  throughput  analysis  for  the  PAP,  and  relaxa¬ 
tion  processing  for  multiple  label  cases.  The  analysis  and  sub¬ 
sequent  testing  is  intended  to  apply  to  monochrome  TV  or  FLIR 
imagery  (hence,  one-dimensional  data)  and  usually  only  one 
object  polarity  (two  labels)  with  extension  to  both  object 
polarities  in  the  same  imagery  (three  labels) . 

7.  Seventh  Quarterly  Report,  January  31,  1980. 

This  report  contains  results  of  relaxation  processing  per¬ 
formed  by  Westinghouse  to  demonstrate  speed,  threshold,  and  con¬ 
vergence  properties  using  test  patterns  and  FLIR  imagery.  This 
evaluation  was  performed  on  the  PDP— VAX  GP  computer  in  preparation 
for  the  processing  of  a  set  of  imagery  on  the  Westinghouse  Pro¬ 
grammable  Array  Processor  (PAP) . 
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